#pandas duplicate records delete

[globosetechnologysolutionspvt]

How to Clean and Preprocess AI Data Sets for Better Results

Introduction

Artificial Intelligence Dataset (AI) models depend on high-quality data to produce accurate and dependable outcomes. Nevertheless, raw data frequently contains inconsistencies, errors, and extraneous information, which can adversely affect model performance. Effective data cleaning and preprocessing are critical steps to improve the quality of AI datasets, thereby ensuring optimal training and informed decision-making.

The Importance of Data Cleaning and Preprocessing

The quality of data has a direct impact on the effectiveness of AI and machine learning models. Inadequately processed data can result in inaccurate predictions, biased results, and ineffective model training. By adopting systematic data cleaning and preprocessing techniques, organizations can enhance model accuracy, minimize errors, and improve overall AI performance.

Procedures for Cleaning and Preprocessing AI Datasets

1. Data Collection and Analysis

Prior to cleaning, it is essential to comprehend the source and structure of your data. Identify key attributes, missing values, and any potential biases present in the dataset.

2. Addressing Missing Data

Missing values can hinder model learning. Common approaches to manage them include:

Deletion: Removing rows or columns with a significant number of missing values.

Imputation: Filling in missing values using methods such as mean, median, mode, or predictive modeling.

Interpolation: Estimating missing values based on existing trends within the dataset.

3. Eliminating Duplicates and Irrelevant Data

Duplicate entries can distort AI training outcomes. It is important to identify and remove duplicate records to preserve data integrity. Furthermore, eliminate irrelevant or redundant features that do not enhance the model’s performance.

4. Managing Outliers and Noisy Data

Outliers can negatively impact model predictions. Employ methods such as

The Z-score or Interquartile Range (IQR) approach to identify and eliminate extreme values.

Smoothing techniques, such as moving averages, to mitigate noise.

5. Data Standardization and Normalization

To maintain uniformity across features, implement:

Standardization: Adjusting data to achieve a mean of zero and a variance of one.

Normalization: Scaling values to a specified range (e.g., 0 to 1) to enhance model convergence.

6. Encoding Categorical Variables

Machine learning models perform optimally with numerical data. Transform categorical variables through:

One-hot encoding for nominal categories.

Label encoding for ordinal categories.

7. Feature Selection and Engineering

Minimizing the number of features can enhance model performance. Utilize techniques such as:

Principal Component Analysis (PCA) for reducing dimensionality.

Feature engineering to develop significant new features from existing data.

8. Data Partitioning for Training and Testing

Effective data partitioning is essential for an unbiased assessment of model performance. Typical partitioning strategies include:

An 80-20 split, allocating 80% of the data for training purposes and 20% for testing.

Utilizing cross-validation techniques to enhance the model's ability to generalize.

Tools for Data Cleaning and Preprocessing

A variety of tools are available to facilitate data cleaning, such as:

Pandas and NumPy, which are useful for managing missing data and performing transformations.

Scikit-learn, which offers preprocessing methods like normalization and encoding.

OpenCV, specifically for improving image datasets.

Tensor Flow and Pytorch, which assist in preparing datasets for deep learning applications.

Conclusion

The processes of cleaning and preprocessing AI datasets are vital for achieving model accuracy and operational efficiency. By adhering to best practices such as addressing missing values, eliminating duplicates, normalizing data, and selecting pertinent features, organizations can significantly improve AI performance and minimize biases. Utilizing sophisticated data cleaning tools can further streamline these efforts, resulting in more effective and dependable AI models. 

For professional AI dataset solutions, visit Globose Technology Solutions to enhance your machine learning initiatives.

[saku-232]

Data Cleansing and Structuring

Data cleansing and structuring are crucial steps in the data preparation process, ensuring that data is accurate, consistent, and formatted for analysis or machine learning. Here's a breakdown of each step:

1. Data Cleansing

This process involves identifying and correcting errors in data, improving its quality. Key activities include:

Removing Duplicates: Identifying and eliminating repeated records in the dataset.

Handling Missing Data: Using methods like imputation (filling missing values), deletion, or flagging to handle null or missing entries.

Correcting Inconsistent Data: Standardizing formats (e.g., date formats, address formats), fixing spelling mistakes, or converting numerical data into the right scale (e.g., removing currency symbols).

Outlier Detection: Identifying and handling data points that deviate significantly from the rest of the data. This might involve removing or correcting them depending on their context.

Noise Filtering: Removing irrelevant or meaningless data that may distort analysis (e.g., stopwords in text data).

2. Data Structuring

Data structuring involves organizing data into a format that is easy to analyze or use for machine learning. This step focuses on making raw data more usable:

Normalization: Scaling features (e.g., values between 0 and 1) to bring them to the same level of magnitude, which helps in various machine learning models.

Encoding: Converting categorical data (e.g., gender, location) into numerical form using techniques like one-hot encoding or label encoding.

Data Aggregation: Combining data from different sources or summary statistics (e.g., sum, average) into a cohesive form.

Feature Engineering: Creating new variables from existing data (e.g., extracting the year from a date field, categorizing data into bins).

Reshaping Data: Converting the dataset into a structured format like tables (e.g., pivoting or unpivoting data, creating time series).

Both of these processes are often done using programming tools like Python (with libraries like pandas, numpy, and scikit-learn for machine learning) or R, and may also involve using SQL for database-related cleaning tasks.

Do you have a specific dataset you're working with, or are you exploring general techniques?

[patel1234565]

Data Cleaning and Preprocessing Techniques in Python

Data cleaning and preprocessing are crucial steps in preparing datasets for analysis or machine learning models. In Python, various techniques streamline the process to ensure accurate and reliable results.

One fundamental step is handling missing data using methods like imputation, where missing values are replaced with appropriate estimates, or deletion, where incomplete observations are removed. Pandas, a popular data manipulation library, provides functions like dropna() and fillna() for these tasks.

Dealing with outliers is another essential aspect. Techniques such as Winsorizing or transforming skewed data using logarithms can help manage extreme values, ensuring they don't unduly influence the analysis.

Normalization and standardization are common preprocessing techniques to scale numerical features, bringing them to a consistent range. Scikit-learn's MinMaxScaler and StandardScaler functions simplify this process.

Categorical variables often require encoding for machine learning models. One-hot encoding or label encoding can be applied using libraries like Scikit-learn or Pandas, ensuring these variables are suitable for algorithmic input.

Handling duplicate data is critical to maintaining the integrity of analyses. Identifying and removing duplicate records can be achieved through Pandas' duplicated() and drop_duplicates() functions.

In summary, mastering data cleaning and preprocessing in Python involves leveraging libraries like Pandas and Scikit-learn for tasks such as handling missing data, managing outliers, scaling features, encoding categorical variables, and removing duplicates. These techniques collectively contribute to a cleaner and more reliable dataset, laying the foundation for robust analyses and machine learning models.

Data science training in pune

[data-science-lovers]

Dealing with Duplicate Rows in Big-Data

[enson1322]

Wrong practice in SEO that can cause Google Penalty

Get insight of a few sick search engine optimization practices which has to keep away from. Penalty, i assume almost every person around the sector recognize the that means of this seven letters word. It refers to as punishment for each leave out-deed or wrong exercise.  Digital Marketing Company Noida As an instance: breaking a pitcher, fighting with classmates is called incorrect deed at school. After doing this instructor offers the punishment in form of stand outside the class, grow to be a chair etc. Similarly google also put penalty on people who are indulge in sick seo practices. Application of 2019 set of rules update google has ensured that folks that have interaction in incorrect search engine optimization practices get penalized. Consequently, today we are going to enlist foremost ill-practices in seo which can cause you penalty. 

Let’s get proper into this: Inbound link manipulation: one way links are one of the ranking requirements on which google price or devalue any web site or web site. The website which has massive variety of one way links ranked better in google search consequences as it generates high quantity of web site visitors. However there are a few webmasters who upload hyperlinks from other web sites to their most important internet site or page in an effort to gain higher rank in google search outcomes. This unwell exercise of manipulating hyperlinks from other sites appeared as incorrect exercise and can reasons you a penalty. 

Doorway pages: Those are the webpages that are purposely built for making fool of seek engine indexes. It approach that it manipulates the hunt engine indexes this exercise is likewise called spam indexing. Those pages do not simplest make idiot of serps however additionally the internet users. Those are that so known as soar pages which redirect you to a few other net page instead of the principle web site/web page. For instance: you had visited the internet web page which has records regarding customer goods like garments, shoes and so on. But in reality that net page only a entice to entice or redirect you on to a few other site including unlawful or ban porn website online. 

Use of skinny content material: It refers to a content which has no delivered cost or significance the form of content material which lies in this category is copied, duplicate, scraped etc. Google panda a tool specifically designed for devaluing the “thin sites” and fee the web sites which have unique or high quality content. The web sites which has too many advertisements, blanks pages and many others. May be devalue on the basis of this element because according to the google eta requirements those sites degrades the excellent of person experience. Car generated content: it refers to content that has already been created and you are the usage of that content in your net page or website. Translated or scraped content from atom rss feeds fall underneath this class. 

The practice of amassing content material from specific pages or web sites and placing them on yours is likewise comes beneath the class of vehicle generated content.  Digital Marketing Services Noida This sick practice can purpose you a penalty and your site/website might be devalued. 

Cellular incompatibility: The fact of 2019 which each site proprietor must recognized or stored in thoughts that cell as a medium of surfing the net is dominating across the globe. So layout and advanced your web site in keeping with the needs and needs of cell customers. Your web site must be flawlessly healthy into cell specs and technicalities which include display screen length, full display view and so on. In case your website online/web site isn't always well suited or applicable to mobile person then it is going to be devalued via google. As a result, it's going to cause the lack of ability internet site visitors to your web site. 

Http compatible:

The 2019 google algorithm replace has made the http (hypertext transfer protocol) as a degree fashionable for ranking the website online and webpages. Its significance has been increased with the release of google “march 2019 middle update”. According to this widespread now the sites and webpages who aren't http like minded can be ranked lower in comparison to the websites that are http well matched. 

Conclusion Above stated practices are comes under the category of terrible seo techniques. Those bad strategies no longer handiest can devalue your webpage or website online however also can suspend or delete your web page completely as an act of breaking google law and violations.

Visit Site:-- https://digitalmar121.weebly.com/blog/what-is-google-adwords

Follows: ---- Youtube, Facebook, Twitter, Linkedin,

[sarahstreetmedia]

Think Your Privacy Is Protected? Think Again.

They hacked my Instagram account last December. My account was down for two weeks, and Instagram was no help. No one ever replied. I went to Facebook and registered as an app developer then requested help for advertising. Whala. Someone called within hours. They restored my account within minutes. But what was the purpose of the hack? Why was my account blocked and not deleted? What happened?

Social media attracts hundreds of millions of users each day. Hacking requires little skill and we can purchase programs like keylogging at Amazon. Malicious coding, spyware, and proprietary information theft are commonly used for simple hacks. With the press of one key, your information is stolen. In my case, I opened an email in my Inbox, the mail looked identical to Instagrams logos, lettering and letterhead. “Your Instagram account has been compromised. Reset your password.” I pressed the link. My account was gone.

Software assures protection, and in most cases even advertise its protection, but If you read the fine print most policies will tell you in the Terms and Conditions they don’t take responsibility for your loss data.

They can buy our personal data for as little as  $0.20, 2018 Current State Of Crime.  Because it sells for so cheap, this makes sellers work harder,  hunting down personal data anyway they can.

Facebook CEO, Mark Zuckerberg is under the microscope for selling major company’s information to Russia. I hope Zuckerberg goes down.

When you download your data from social media platforms, you’ll probably be shocked to discover what they have stored. Think you deleted it years ago? Think again. Nothing ever gets deleted. If someone hacked your account, they have access to everything you have ever typed, including what you sent via private message and your previous passwords.

1 out of 5 files aren’t protected. Your software device firewall and added software security are not protecting you entirely. When you install an app, it asks for permission to gain access in your device. And that is exactly what they have, access.

Cybercrime is more profitable than drug trade.

Despite the payout of personal data, cybercrime is incredibly expensive and the most expensive part is loss. In 2015 cybercrime caused a whopping 3 trillion in damages and it’s estimated to cause 6 trillion in damages by 2021.  When our credit reporting agencies and emails are hacked, this number increases at an enormous rate and speed.

We may think it’s bigger companies to blame for Cyber breaches, but in fact 95% of security failures  are the customers fault.

Microsoft Office file formats (Excel, Word, Power point), compromise the most prevalent group of malicious file extensions.  Making up 38% of Cyber attacks, effecting you daily. These are commonly sent as extensions or attachments in emails and are utilized by most people. These Microsoft file formats are vectors for malware and are easily overlooked.

WordPress is a free and open source website tool; popular software that’s powering nearly half of all websites on the Internet. But 73.2% of all Wordpress installations are open to vulnerabilities.

Crypto Hacking attacks are up by 8,500% yet the unemployment rate for Cyber security jobs are approaching 0%. Reaching 3.5 million needed jobs to fill by 2021.  This market is growing too fast for the world to keep up.

Androids are at extreme risk. Malicious software that is installed on android platforms has increased by 400%. People tend to be more relaxed with their cell phones, probably because most of us carry it with ourselves. Only 50% buy added protected on their smartphones and 72% on their laptops and desktops computers.  It’s  likely that our smartphones store more data than our desktops and laptops. In a hyper-connected world the majority of us use our phones for everything. We have become overly connected and have developed a deep loving relationship with our phones. We tend to trust it, it’s our safety for so many reasons. Therefore, we  input more personal information on our phones than on our computers.

Over half of China’s electronic devices are infected. China is the #1 country with the most infected malware. There are billions of malware samples, “PandaLabs, the laboratory of Panda Security, detected and neutralized more than 84 million new malware samples throughout 2015.” Last year there were 304 million samples detected in the world. It’s estimated that over 230,000 new malware samples are produced daily in China and roughly 1 million new threats are released every day in the world, CNN Business, 2014.

It can take seconds to steal your data. Below is a list of malware and hacking techniques to beware of.

Digital extortion:  one of the most lucrative ways cybercriminals can profit. Hackers steal files or photos from a victim’s computer and demand a ransom in exchange for a key to decrypt their files. The bulk of digital extortion is done with ransomware attacks. Other types include, phishing, negative review and spamming (the cybercriminal will retract the negative review for cost), and blackmail.

Common ploy: Send 300k in Bitcoin or the cybercriminal will release personal and private information about you. They may even have a video of you doing someone that makes you feel guilty. Or they may threaten to have a video of you or know something about you doing something that makes you feel guilty to cave in. They will most likely provide your accurate banking information and personal information to validate that they are in fact real hackers to scare you. Therefore the blackmail works so well. Especially if you have kids and loved ones you want to protect. What should you do? Save yourself 300k in Bitcoin. Change your passwords and bank information. Don’t entertain them.

Guilt has made many people prosper.

If you are computer savvy, check your terminal and look for anything unusual. You can open Terminal from Utilities. Type ‘man’ followed by a space and then the process name, e.g. “man nefwork.

Social media scams. Users do all the work for the cybercriminal just by sharing a video or story on sketchy sites. The virus spreads rapidly because people are more likely to click on something posted by a friend.

Likejacking and clickjacking are common social media spam hoaxes made in an attempt to steal your private information. A common ploy, “Facebook will donate $1 toward the child’s care for every like.” Cyber thieves will say anything to convince the reader to hit the link to embed the malware in your device.

Proxy hijacking is when a hacker creates a copy of the victims web page on a proxy server, using keyword stuffing techniques, linking the original site to a copied website hoping to increase search engine rankings. The victims site will then rank lower. Search engines like Google will see the victims site as a duplicated website, then Google might remove it from its index.

Keylogging is common in the workforce and keylogging software is available on Amazon. Keylogs record your information by recording your keystrokes. Information is gathered and collected over time.  If you keep getting the same email threat, chances are, your keystrokes are being recorded. Keylogs are also commonly used in the home, it’s a common a program purchased for someone with trust issues.

Malware is hard to detect. Its intention is to cause damage to a device, server, or network.

Trojan is a computer program which misleads users convincing them the content is true.

Spyware  Like keyloggers, this software aims to gather information by recording your keystrokes and activities. The information can be sent to another entity without the consumer’s knowledge.  A common example is when a cybercriminal waits for the user to install an update. Then they have access to your devices.

A crawler or hunter is not malware or a virus, instead it wreaks havoc on your device and disrupts browsing activity.

Phishing. 1 in 3 emails are phishing. “Roughly 12 percent are victims of the actual infections that result from phishing.”

What can you do? Educate yourself on keyloggers, Trojans, spyware and malware. Protect yourself against malicious apps and third-party tools. Beware of apps and software asking permission for sms, microphone access, and device permission. Beware of fraudulent email accounts. Hackers use email names from companies we recognize then imitate a company’s name to look similar: @gmail.com, @gmall.com.

Never download from 3rd party sites, change passwords weekly and Include specialized characters, capital letters and numbers.  Never use the same password for over one account. When you receive a text asking to validate a pin, keep in mind, it might be a hack.  Instead, use an old unactivated phone for two factor authentications for extra protection.

When you open a suspected email,  check the URL on your desktop. An unsafe URL starts with http:// a protected URL starts with https:// Know that there are ways to manipulate a URL. so it looks like it is a trusted company.  Also know that there are sites you go incognito with to get a URL checked out.

Remember that open networks and WEP security, aren’t secure at all.  Consider changing your proxy when using Wi-Fi networks that are shared. Change or hide your IP address. Hackers can exploit your IP address. But they can’t do much if your firewall is on, your firewall stops them from entering your programs, they can get into your ports. Your personal details and your IP address are not connected, so that is false, your IP does not provide information about you. Consider using an extra hardware firewall for added protection and block your IP address. 

Some sites will prevent your account from logging in if you use a different IP address then what you normally used to log in. I can easily unblock mine when logging into sites with difficulty.  Working in social media, I find a lot of third-party sites change their IP for privacy and assign one specifically to you. It’s also good to change your proxy settings to prevent proxy hijacking.

Unfortunately, it’s impossible to avoid all hacks. But we can educate ourselves on what we can control and protect ourselves.

Sarah Simplot

March 10, 2019

[onenationprinc-blog]

From pranks to nuclear sabotage, this is the history of malware

Since the dawn of modern computing, software has been as capable as the programmers who created it. Their intentions became its capabilities, and that’s brought us a world of wondrous and powerful applications across a wide variety of platforms and mediums. Along the way, it’s also lead to the creation of incredibly malicious, and in some cases downright dangerous, software. We are, of course, talking about malware. We’ve all come across malware at some point. You might’ve been spammed during the heyday of adware and popups, faced off against a nasty trojan that tried to steal your identity, or even dealt with a system-paralyzing piece of blackmailing ransomware. Today, millions upon millions of unique programs are designed to target your system, your files, and your wallet. While they all have different footprints and trajectories, they all have their roots in humble beginnings. To understand malware, you must return to the digital primordial soup that would one day evolve into the millions of nefarious programs we face off against today. This is the history of malware, and of the techniques used over decades to combat it. An innocent birth The modern world faces criminal and nation state hacking that could threaten everyone’s way of life. Yet the early days of malware were free of malice. Back then, the intention was to see what was truly possible with computing, not to harm, steal, or manipulate. The idea for a virus, or a self-replicating string of code, was first coined by computing visionary John Von Neumman. In 1949, he postulated the potential for a “self-reproducing automata” that would be able to pass along its programming to a new version of itself. ‘I’m the Creeper:Catch me if you can.’ The first known recorded instance of a computer virus was the Creeper Worm, developed by Robert H. Thomas in 1971. The first iteration of Creeper couldn’t clone itself, but it was able to move from one system to another. It would then display the message, ‘I’m the Creeper: Catch me if you can.’ While it seems likely the first self-replicating code and its creator are lost, the first recorded instance of such software is the Creeper Worm, developed by Robert H. Thomas in 1971 at BBN Technologies. Creeper ran on the TENEX operating system and was impressively sophisticated for its time. Unlike many of its successors, which would require physical mediums to spread their payloads, Creeper was able to move between DEC’s PDP-10 mainframe computers over the earliest iteration of the ARPANET, a progenitor network of the internet the world would come to adopt in later years. The first iteration of Creeper couldn’t clone itself, but it was able to move from one system to another. It would then display the message, “I’m the Creeper: Catch me if you can.” A new version of Creeper was later created by Thomas’ colleague at BBN Technologies, Ray Thomlinson – better known as the inventor of email. It did duplicate itself, leading to an early understanding of the problem such viruses, or worms, could cause. How do you control them once you send them off? In the end, Thomlinson created another program called Reaper, which moved around the network and deleted any copies of Creeper it found. Thomlinson didn’t know it, but he had created the very first piece of anti-virus software, starting an arms race between hackers and security professionals that continues to this day. Creeper, although mocking in its message, was not designed to cause problems for the system. Indeed, as Thomlinson himself explained to computing historian, Georgei Dalakob, “The creeper application was not exploiting a deficiency of the operating system. The research effort was intended to develop mechanisms for bringing applications to other machines with intention of moving the application to the most efficient computer for its task.” Peaks and Troughs In the years that followed the proliferation and subsequent deletion of the Creeper virus from those ancient mainframe systems, a few other pieces of malware appeared and iterated upon the idea. The self-replicating Rabbit virus was created by an unknown – but supposedly, very much fired – programmer in 1974, and was followed shortly afterwards by the Animal virus, which took the form of a quiz game. Malware creation then went through one of its periodic developmental droughts. But that all changed in 1982, when Elk Cloner made its appearance, and a new wave of viruses began to rise. “With the invention of the PC, people started writing boot sector viruses that were spread on floppies,” Zone Alarm’s Skyler King told Digital Trends. “People who were pirating games or sharing them on floppies [were being infected].” Elk Cloner was the first to use that attack vector, though it was completely benign, and not thought to have spread far. Its mantle was picked up four years later by the Brain virus. That piece of software was technically an anti-piracy measure created by two Pakistani brothers, though it had the effect of making some infected disks unusable due to timeout errors. “Those were kind of the first viruses as we would consider them,” King said. “And they were propagating so that if you put in a floppy, they could copy to it, and spread that way.” The change in attack vector was noteworthy, because targeting a system from a different angle would become the hallmark of new malware in the years that followed. “Things kind of shifted over to the Unix side with the mainstream use of the internet and universities, like the Morris worm in November 1988,” King continued. “That was interesting, because the Morris worm was [written by] the son of the head of the NSA […] He found a flaw in two protocols that were used in Unix. The flaw in SMTP, the mail protocol that allowed you to send email, [was used to] propagate it, and within a day it took down the internet as it existed in 1988.” The Morris worm was said to be originally designed to map the internet, but it bombarded computers with traffic, and multiple infections could slow them to a crawl. It is ultimately credited with bringing down around 6,000 systems. Robert Morris, the worm’s creator, became the first person ever tried under the Computer Fraud and Abuse Act of 1986. He was sentenced to three years of probation and fined $10,050. Today, Morris is an active researcher of computer network architectures and tenured professor at MIT. The Morris Worm became the proof of concept for a variety of other pieces of malware from that same period, all of which targeted boot sectors. It started the next wave in virus development. Many variants of that idea were collected under the “Stoned,” label, with notable entries like Whale, Tequila, and the infamous Michelangelo, which annually created panic in organizations with infected systems. The last days of summer For the first decades of their existence, even the prolific and damaging viruses were of relatively benign design. “They were just people having fun trying to get street cred in the underground scene to show what they could do,” King told Digital Trends. Defensive methods were still far behind the virus writers, however. Even simple malware like the ILoveYou Worm — which made its appearance in the year 2000 — could cause unprecedented damage to systems worldwide. Malwarebytes‘ VP of technology, Pedro Bustamante, remembers it well. “It was a visual basic script that was a mass mailer that would auto-attach a script, and the [anti-virus firms] weren’t ready to do a lot of script based detection back then,” he said. Filipino programmer Onel de Guzman is most often credited with the worm’s creation, though he has always denied developing its attack vector, and suggests that he may have released the worm by accident. Some rumors suggest the real culprit behind its creation was a friend of his, Michael Buen, who tricked Guzman into releasing it because of a love rivalry. The ILoveYou Worm caused over $15 billion in damage globally. “We were on lockdown at Panda labs for like three days for that one. People didn’t sleep.” “We were on lockdown at Panda labs for like three days for that one,” Bustamante continued. “People didn’t sleep. That was the epicenter of that script kiddie movement where anyone could create a script and make a mass mailer and it would have a huge propagation. Massive number of infections. That was typically only possible with an advanced network worm back in the day.” Zone Alarm’s King faced similarly sleepless nights with some other malware spreading across the growing internet during that time, citing the likes of Code Red and SQL Slammer as particularly problematic. While worms and viruses had security experts pulling their hair out, and company executives scared of the millions or billions of dollars of damage they were doing, nobody knew that the malware wars were only just getting started. They were about to take a dark and dangerous turn. No longer a game As internet use grew, advertising networks started to earn money online, and dot-coms raked in investor cash. The internet transformed from a small community known by few into a widespread, mainstream avenue of communication, and a legitimate way to make millions of dollars. The motive for malware followed, shifting from curiosity to greed. Kaspersky Cyberthreat real-time map shows cyberattacks taking place right now throughout the world. “When more people started using the internet and people were looking at ads online and companies were out there making money on ad clicks, that’s when you started seeing the rise of adware and spyware,” King continued. “You started to see viruses that ran on individual computers that sent out spam to try and buy into products, or adware that used clickfraud that showed ads for things so that it would simulate you clicking on the link, so they’d make money.” Organized crime soon realized that clever programmers could make established underground enterprises a lot of money. With that, the malware scene turned several shades darker. Prepackaged malware kits created by criminal organizations began to appear online. Famous ones like MPack were ultimately used to infect everything from individual home systems, to banking mainframes. Their level of sophistication, and link to real-world criminals, up the stakes for security researchers. “That’s when we started seeing some of the gangs that were behind some of these more modern attacks and malware. It was scary.” “We discovered MPack at Panda Security, and we did an investigation and a big paper that was all over the news,” Malwarebytes’ Bustamante explained. “That’s when we started seeing some of the gangs that were behind some of these more modern attacks and malware. It was scary. Most researchers at Panda said that they didn’t want their name anywhere near the report.” But the report was released, and it highlighted how deeply malware and organized criminal gangs had become. “It was a lot of Russian gangs. We had pictures of their gatherings. It was like a company,” Bustamante said. “They had people doing marketing, executives, company get togethers, competitions for programmers who wrote the best malware, tracking affiliates, they had everything. It was amazing. They were making more money than we were.” That money was shared with talented programmers, ensuring the organizations attracted the best talent they could. “We started seeing pictures of mafia looking guys from Eastern Europe giving away fancy cars to the programmers, and suitcases full of money,” he said. Vulnerabilities exploited The pursuit of profit lead to more sophisticated malware and new attack vectors. The Zeus malware, which appeared in 2006, used basic social engineering to trick people into clicking email links, ultimately letting the creator steal victims’ login information, financial details, PIN codes, and more. It even facilitated so-called “man in the browser,” attacks, where malware can request security information at the point of login, harvesting even more information from victims. News clips showing various malware through the years. Those creating malware also learned they didn’t have to use the software themselves, and could simply sell it to others. The MPack kit Bustamante came across at Panda Security in the mid ’00s was a perfect example. It was updated month to month since its early creation, and regularly resold. Even the alleged author of Zeus, Russian-born Evgeniy Mikhailovich Bogachev, began to sell his malware, before handing off control of the Zeus malware platform to another programmer. He’s still at large today. The FBI has a bounty on information leading to Bogachev’s arrest, offering as much as $3 million to anyone who can help catch him. By 2007, more malware was being created every year than had existed in the entire history of malware, and each new mass attack fueled the fire. Selling pre-packaged malware the way that Bogachev did marked another shift in malware creation. Now that malware could be used to make money, and virus writers could make money selling it as a tool, it became more professional. Malware was crafted into a product, commonly termed an exploit kit. “It was really sold as a business,” Zone Alarm’s King told Digital Trends. “They [offered] support, software updates to the latest exploits, it was pretty amazing.” By 2007, more malware was being created every year than had existed in the entire history of malware, and mass attacks on the ever-growing number of computers drove business. This spurred the rise of large-scale botnets which were offered for rent to those wishing to conduct denial of service attacks. But end-users could only be tricked into clicking links for so long. As they became more educated, the exploit kits and their authors needed to evolve again. “[Malware writers] had to come up with a way to install the threat automatically,” MalwareBytes CEO Marcin Kleczynski told Digital Trends. “That’s where the exploit techniques, social engineering, and macros in Powerpoint and Excel started getting way more [sophisticated].” MalwareBytes CEO Marcin Kleczynski. MalwareBytes Fortunately for the malware authors, websites and offline software began to adopt Web 2.0 principles. User interaction and complex content creation were becoming far more prevalent. To adapt malware writers started targeting Internet Explorer, Office applications, and Adobe Reader, among many others. “The more complex software gets, the more it can do, the more engineers working on it […] the more mistake prone that software is and the more vulnerabilities you’ll find over time,” Kleczynski said. “As software gets more complex and Web 2.0 happened, and Windows kept evolving, it got more complex and more vulnerable to the outside world.” By 2010, it seemed that not-for-profit malware had all but died out, with for-profit being the near-exclusive motivation for crafting it. That, it turned out, was wrong. The world abruptly learned that organized crime was nothing compared to the most dangerous malware, crafted in secret by nations. Digital warfare The first example of a nation flexing its military might online was the Aurora attack on Google. The search giant, long standing as one of the world’s most prominent digital entities, found itself under sustained attack at the close of 2009 by hackers with ties to the Chinese Liberation Army. When the rest of the world learned about it in January 2010, it marked a turning point in what experts realized malware, and its authors, were capable of. How Stuxnet Worked K-Lopa/IEEE Spectrum The attack targeted dozens of high-level tech firms like Adobe, Rackspace, and Symantec, and were thought to be an attempt to modify the source code of various software suites. Later reports suggested it was a Chinese counterintelligence operation to discover U.S. wiretap targets. As ambitious and impressive as that attack was, however, it was surpassed just months later. “The cat really came out of the bag with Stuxnet,” Bustamante told Digital Trends. “Before that […] you could see it in certain attacks and in the things like Pakistan, India internet being cut down undersea, [but] Stuxnet is where the shit hit the fan, and everyone started freaking out.” “Chaining together several zero-day vulnerabilities [in Stuxnet], really advanced targeting of specific nuclear facilities. It’s amazing. It’s the type of stuff that you would only see in a novel.” Stuxnet was built to sabotage Iran’s nuclear program, and it worked. Even now, eight years after its appearance, security professionals speak of Stuxnet with a tone of awe. “Chaining together several zero-day vulnerabilities, really advanced targeting of specific nuclear facilities. It’s amazing,” Bustamante said. “It’s the type of stuff that you would only see in a novel.” Kleczynski was just as impressed. “[…] if you look at exploits being used for an offensive cyber-security capability, it was a pretty damn good one. The [way it went after the] Siemens programmable logic computers? It was beautifully architected to destroy the centrifuges.” Although no one claimed responsibility for Stuxnet in the years that followed, most security researchers think it the work of a combined U.S.-Israeli taskforce. That only seemed more likely when other revelations, like NSA hard drive firmware hacking, showed the true potential of nation state hackers. The Stuxnet style of attack would soon become commonplace. Exploit kits continued to be a major attack vector in the years that followed, but as Bustamante told us in our interview, zero-day vulnerabilities chained together are now something that Malwarebytes and its contemporaries see every day. That’s not all they see. There’s a new phenomenon with origins that can be traced almost back to the start of our story. It has caused no end of trouble as of late, and may well do so into the future. Your money or your files The very first ransomware attack technically happened as far back as 1989, with the AIDS Trojan. Sent out to AIDS researchers on an infected floppy disc, the malware would wait for the system to be booted 90 times before encrypting files and demanding a payment of $189 in cash, sent to a PO Box address in Panama. Although that piece of malware was called a trojan at the time, the idea of forcibly obfuscating files, denying a user access to their own system, and demanding some form of payment to return it to normal, became the key components of ransomware. It began to resurface again in the mid-00s, but it was the growth of anonymous cryptocurrency Bitcoin that made ransomware common. “If you infect someone with ransomware and ask them to deposit into a bank account, that account is going to get closed down pretty quick,” Zone Alarm’s King explained. “But if you ask someone to deposit some bitcoin in a wallet, the consumers pay. There’s really no way to stop it.” Ransomware developers make it easy for victims to purchase cryptocurrency and send it to them. Considering how difficult it is to regulate bitcoin in everyday functions with legitimate uses, it makes sense that stopping it from being leveraged by criminals is even more so. Especially since people pay the ransoms. Just as with the exploit kits and the corporate structure that backs them, ransomware developers make it as easy as possible for victims to purchase cryptocurrency and send it to them. But in the latter half of the teen years of the 21st century, we’ve started to see further evolution of these tactics, as once again those writing the malicious software have followed the money. “What’s surprised me with ransomware is how quickly it went from you and I, to our companies,” Kleczynski said. “A year or two ago it was us who were getting infected, not Malwarebytes, not SAP, Oracle and so on. They’ve clearly seen the money and companies are willing to pay it.” What’s next? For most of the experts we spoke to, ransomware continues to be the big threat they’re concerned with. Zone Alarm’s King was keen to talk about his company’s new anti-ransomware protections and how businesses needed to be aware of how dangerous the tactic was. Kleczynski sees it as a hugely profitable model for malware writers, especially when you bring in the rise of infected Internet of Things devices, which have made up some of the largest botnets the world has ever seen. Timelapse of a DDoS attack that took place in 2015 on Christmas Day. Using British Airways’ website as an example, he asked the rhetorical question of how much it would be worth it for that company to maintain its online ticketing system if threatened. Would such a company be willing to pay an extorter $50,000 if its website were to go down for even a few hours? Would it pay $10,000 at the mere threat of such an action? With the potential to lose millions in sales, or even billions in market value should stock prices react to such an attack, it’s not hard to imagine a world where that’s a regular occurrence. To Kleczynski, this is just the old world finally catching up with the new. It’s the organized crime tactics of yesteryear being applied to a modern world. “Today, it’s ‘would you like to purchase some ransomware insurance? It’d be a shame if your website went down for 24 hours.’” “This used to just be racketeering. ‘Would you like to purchase some fire insurance? It would be a shame if something happened to your building,’” he said. “Today, it’s ‘would you like to purchase some ransomware insurance? It’d be a shame if your website went down for 24 hours.’” That criminal involvement still scares MalwareBytes’ Bustamante, who tells us that the company regularly sees threats to its developers hidden in malware code. As concerned as he and the company are about their own personal safety though, he sees the next wave as something more than just ransomware. He sees it as an assault on our ability to perceive the world around us. “If you ask me what the next wave is, it’s fake news,” he said. “Malvertising has moved on […] it’s now clickbait and fake news. Disseminating this kind of news is the name of the game and it’s going to be the big next wave.” Considering how involved nation states appear to have been in that practice themselves in recent years, it’s hard to imagine he’s wrong. As threatening as malware attacks from organized crime, government-sponsored vigilantes, and militarized hackers are, the most reassurance you can take in such a time of uncertainty is that the weakest link in the security chain is almost always the end user. That’s you.. It’s scary, but empowering, too. It means that although the people writing the malware, the attack vectors and the very reason for creating viruses and trojans in the first place may have changed, the best ways of staying safe online are the old ways. Keep strong passwords. Patch your software. And be careful what links you click. As Malwarebytes Klecyzinski told us after our interview, “If you’re not paranoid, you’re not going to survive.” Editors’ Recommendations https://www.digitaltrends.com/computing/history-of-malware/ https://blog.cyberprosocial.com/2018/03/29/from-pranks-to-nuclear-sabotage-this-is-the-history-of-malware/

[sarahstreetmedia]

Think Your Privacy Is Protected? Think Again.

My Instagram account, @sarahstreetmedia was hacked last December. My account was down for two weeks, and Instagram was no help. No one ever replied. I went to Facebook and registered as an app developer then requested help for advertising. Whala. Someone called within hours. They restored my account within minutes. But what was the purpose of the hack? Why was my account blocked and not deleted? What happened?

Social media attracts hundreds of millions of users each day. Hacking requires little skill and we can purchase programs like keylogging at Amazon. Malicious coding, spyware, and proprietary information theft are commonly used for simple hacks. With the press of one key, your information is stolen. In my case, I opened an email in my Inbox; the mail looked identical to Instagrams logos, lettering and letterhead. “Your Instagram account has been compromised. Reset your password.” I pressed the link. My account was gone.

Software assures protection, and in most cases even advertise its protection, but If you read the fine print most policies will tell you in the Terms and Conditions they don’t take responsibility for your loss data.

Our personal data can be bought for as little as  $0.20, 2018 Current State Of Crime.  Because it sells for so cheap, this makes sellers work harder,  hunting down personal data anyway they can.

Facebook CEO, Mark Zuckerberg is under the microscope for selling major company’s information to Russia. I personally hope Zuckerberg goes down.

When you download your data from social media platforms, you’ll probably be shocked to discover what it has stored. Think you deleted it years ago? Think again. Nothing ever gets deleted. If someone hacked your account, they have access to everything you have ever typed, including what you sent via private message and your previous passwords.

1 out of 5 files aren’t protected. Your software device firewall and added software security are not protecting you entirely. When you install an app, it asks for permission to gain access in your device. And that is exactly what they have access.

Cybercrime is more profitable than drug trade.

Despite the payout of personal data, cybercrime is incredibly expensive and the most expensive part is loss. In 2015 cybercrime caused a whopping 3 trillion in damages and it’s estimated to cause 6 trillion in damages by 2021.  When our credit reporting agencies and emails are hacked, this number increases at an enormous rate and speed.

We may think it’s bigger companies to blame for Cyber breaches, but in fact 95% of security failures  are the customers fault.

Microsoft Office file formats (Excel, Word, Power point), compromise the most prevalent group of malicious file extensions.  Making up 38% of Cyber attacks, effecting you daily. These are commonly sent as extensions or attachments in emails and are utilized by most people. These Microsoft file formats are vectors for malware and are easily overlooked.

WordPress is a free and open source website tool; popular software that’s powering nearly half of all websites on the Internet. But 73.2% of all Wordpress installations are open to vulnerabilities.

Crypto Hacking attacks are up by 8,500% yet the unemployment rate for Cyber security jobs are approaching 0%. Reaching 3.5 million needed jobs to fill by 2021.  This market is growing too fast for the world to keep up.

Androids are at extreme risk. Malicious software that is installed on android platforms has increased by 400%. People tend to be more relaxed with their cell phones, probably because most of us carry it with ourselves. Only 50% buy added protected on their smartphones and 72% on their laptops and desktops computers.  It’s  likely that our smartphones store more data than our desktops and laptops. In a hyper-connected world the majority of us use our phones for everything. We have become overly connected and have developed a deep loving relationship with our phones. We tend to trust it, it’s our safety for so many reasons. Therefore, we input more personal information on our phones than on our computers.

Over half of China’s electronic devices are infected. China is the #1 country with the most infected malware. There are billions of malware samples, “PandaLabs, the laboratory of Panda Security, detected and neutralized more than 84 million new malware samples throughout 2015.” Last year there were 304 million samples detected in the world. It’s estimated that over 230,000 new malware samples are produced daily in China and roughly 1 million new threats are released every day in the world, CNN Business, 2014.

It can take seconds to steal your data. Below is a list of malware and hacking techniques to beware of.

Digital extortion:  one of the most lucrative ways cybercriminals can profit. Hackers steal files or photos from a victim’s computer and demand a ransom in exchange for a key to decrypt their files. The bulk of digital extortion is done with ransomware attacks. Other types include, phishing, negative review and spamming (the cybercriminal will retract the negative review in exchange for cost), and blackmail.

Common ploy: Send 300k in Bitcoin or the cybercriminal will release personal and private information about you. They may even have a video of you doing someone that makes you feel guilty. Or they may threaten to have a video of you or know something about you doing something that makes you feel guilty to cave in. They will most likely provide your accurate banking information and personal information to validate that they are in fact real hackers to scare you. This is why the blackmail works so well. Especially if you have kids and loved ones you want to protect. What should you do? Save yourself 300k in Bitcoin. Change your passwords and bank information. Don’t entertain them.

Guilt has made many people prosper.

If you are computer savvy, check your terminal and look for anything unusual. You can open Terminal from Utilities. Type ‘man’ followed by a space and then the process name, e.g. “man nefwork.

Social media scams. Users do all the work for the cybercriminal just by sharing a video or story on sketchy sites. The virus spreads rapidly because people are more likely to click on something posted by a friend.

Likejacking and clickjacking are common social media spam hoaxes made in an attempt to steal your private information. A common ploy, “Facebook will donate $1 toward the child’s care for every like.” Cyber thieves will say anything to convince the reader to hit the link to embed the malware in your device.

Proxy hijacking is when a hacker creates a copy of the victims web page on a proxy server, using keyword stuffing techniques, linking the original site to a copied website with the hopes of increasing search engine rankings. The victims site will then rank lower. Search engines like Google will see the victims site as a duplicated website, then Google might remove it from its index.

Keylogging is common in the workforce and keylogging software is available on Amazon. Keylogs record your information by recording your keystrokes. Information is gathered and collected over a period of time.  If you keep getting the same email threat, chances are, your keystrokes are being recorded. Keylogs are also commonly used in the home, it’s a common a program purchased for someone with trust issues.

Malware is hard to detect. Its intention is to cause damage to a device, server, or network.

Trojan is a computer program which misleads users convincing them the content is true.

Spyware  Like keyloggers, this software aims to gather information by recording your keystrokes and activities. The information can be sent to another entity without the consumer’s knowledge.  A common example is when a cybercriminal waits for the user to install an update. Then they have access to your devices.

A crawler or hunter is not malware or a virus, instead it wreaks havoc on your device and disrupts browsing activity.

Phishing. 1 in 3 emails are phishing. “Roughly 12 percent are victims of the actual infections that result from phishing.”

What can you do? Educate yourself on keyloggers, Trojans, spyware and malware. Protect yourself against malicious apps and third-party tools. Beware of apps and software asking permission for sms, microphone access, and device permission. Beware of fraudulent email accounts. Hackers use email names from companies we recognize then imitate a company’s name to look similar: @gmail.com, @gmall.com.

Never download from 3rd party sites, change passwords weekly and Include specialized characters, capital letters and numbers.  Never use the same password for over one account. When you receive a text asking to validate a pin, keep in mind, it might be a hack.  Instead, use an old unactivated phone for two factor authentications for extra protection.

When you open a suspected email,  check the URL on your desktop. An unsafe URL starts with http:// a protected URL starts with https:// Know that there are ways to manipulate a URL. so it looks like it is a trusted company.  Also know that there are sites you go incognito with to get a URL checked out.

Remember that open networks and WEP security, aren’t secure at all.  Consider changing your proxy when using Wi-Fi networks that are shared. Change or hide your IP address. Hackers can exploit your IP address. But they can’t do much if your firewall is on, your firewall stops them from entering your programs, they can get into your ports. Your personal details and your IP address are not connected, so that is false, your IP does not provide information about you. Consider using an extra hardware firewall for added protection and block your IP address. I personally use Express VPN.

Some sites will prevent your account from logging in if you use a different IP address then what you normally use to log in. I can easily unblock mine when logging into sites with difficulty.  Working in social media, I find a lot of third-party sites change their IP for privacy and assign one specifically to you. It’s also good to change your proxy settings to prevent proxy hijacking.

Unfortunately, it’s impossible to avoid all hacks. But we can educate ourselves on what we can control and protect ourselves. 

sarah@withinaframe.com | www.sarahstreetmedia.com | @sarahstreetmedia